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Abstract 

Effective  and  efficient  DoD  acquisition  programs  require  the  analysis  of  a  wide  range 
of  materiel  alternatives.  Alternative  diversity,  difficulties  in  selecting  metrics  and  measuring 
performance,  and  other  factors  make  the  Analysis  of  Alternatives  (AoA)  difficult.  The 
benefits  of  alternatives  should  be  included  in  AoA,  but  cost  estimates  predominate  most 
AoA  processes.  Incorprating  benefits  into  AoA  is  particularly  difficult  because  of  the 
intangible  nature  of  many  important  benefits.  The  current  work  addresses  the  need  to 
improve  the  use  of  benefits  in  AoA  by  building  a  system  dynamics  model  of  a  military 
operation  and  integrating  it  with  the  Knowledge  Value  Added  (KVA)  methodology.  The 
synergies  may  be  able  to  significantly  improve  the  accuracy  of  KVA  estimates  in  the  AoA 
process.  A  notional  mobile  weapon  system  was  modeled  and  calibrated  to  reflect  four 
weaponized  Unmanned  Aerial  Vehicles  (UAV).  Modeling  a  hypothetical  AoA  for  upgrading 
one  of  the  UAV  indicated  that  there  were  potentially  significant  synergies  that  can  increase 
the  number  of  alternatives  that  could  be  analyzed,  establishing  common  units  of  benefit 
estimates  for  an  AoA,  improved  reliability  of  an  AoA,  and  improved  justification  of  AoA 
results.  These  can  improve  alternative  selection,  thereby  improving  final  materiel 
effectiveness,  thereby  improving  DoD  acquisition  processes. 
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Introduction 


The  US  defense  acquisition  process  is  initiated  by  the  Joint  Capabilities  Integration 
and  Development  System  (JCIDS),  which  is  one  of  the  three  major  decision  support 
systems  used  in  the  DoD  to  interconnect  and  arrive  at  new  warfighting  capability.  The  JCIDS 
formulates  force  requirements  with  a  “top  down”  approach  that  serves  as  both  a  Joint  Force 
integrative  process  and  one  that  can  also  hierarchically  decompose  the  complexities  of  the 
battle  spaces  and  their  critical  mission  elements.  It  must  also  be  aligned  with  the  Planning, 
Programming,  Budgeting  and  Execution  System  (PPBES)  funding  process  as  a  way  to 
descend  from  the  strategic  to  the  tactical  in  acquisition  programs  and  budgets  ( CJCSI 
3170. 01G,  2009).  JCIDS  uses  Capability  Based  Assessments  (CBA)  to  validate  capability 
gaps,  to  discover  solutions  such  as  those  addressed  by  non-materiel-type  changes  to 
doctrine,  organization,  training,  leadership  &  education,  personnel,  or  facilities,  or  to  pursue 
materiel  solutions.  Essential  to  CBA  subprocesses  is  the  knowledge  of  various  functional 
warfighting  Joint  Capability  Areas  and  how  those  communities  operate  within  a  joint 
paradigm.  Functional  Capability  Boards  are  organized  around  top-tier  functional  areas  such 
as  Force  Application,  Logistics,  etc. 

Once  needs  are  specifically  derived  in  an  area,  and  it  is  ascertained  that  they  can 
only  be  addressed  by  new  materiel,  the  acquisition  community  is  still  often  left  with  either  a 
variety  of  system  types  or  technical  approaches  within  a  particular  system  type  to  fully 
address  that  capability  need.  An  in-depth  Analysis  of  Alternatives  (AoA)  helps  sponsors  and 
program  managers  compare  options.  Examples  include  manned  or  unmanned  aircraft 
versus  a  missile,  chemical  energy  versus  kinetic  energy  kill  mechanisms,  etc.  In  the  past, 
these  have  also  been  called  Cost  and  Operational  Effectiveness  Analyses,  and  cost- 
effectiveness  analyses — all  variants  of  a  Business  Case  Analysis. 

The  focus  of  the  current  work  is  on  improving  the  Analysis  of  Alternatives  (AoA) 
process  that  is  used  to  make  these  major  acquisition  decisions.  We  will  demonstrate  the  use 
of  a  system  dynamics  modeling  approach  that  incorporates  common  units  of  benefits 
parameters  using  the  KVA  methodology  and  the  potential  improvements  that  might  result  in 
an  AoA. 

Problem  Description 

In  typical  weapon  system  acquisition  programs,  there  is  a  point  where  an  AoA  is 
conducted  to  select  the  most  viable  and  cost-effective  materiel  solution,  so  that  it  may  be 
pursued  into  advanced  development  and  production.  Often,  a  selection  for  advanced 
technical  development  among  competitive  system  prototypes  is  needed — a  practice  that  has 
recently  become  official  DoD  acquisition  policy,  but  is  not  a  new  idea  (USD(AT&L)  memo, 
September  19,  2007,  Subj:  Prototyping  and  Competition). 

System  concepts  are  further  clarified  during  the  Materiel  Solutions  Analysis  phase 
and  the  accompanying  AoA  process.  As  part  of  these  processes,  programs  move  toward 
several  kinds  of  evaluative  cost  comparisons  that  formulate  costs  estimates  across  a 
notional  life  cycle.  In  these  early  stages,  programs  use  analogous  and  parametric  cost 
estimation  techniques.  Parameters  of  system  performance  and  key  system  characteristics 
are  selected  from  technical  and  operational  inputs.  Usually,  several  Key  Performance 
Parameters  included  in  the  Initial  Capability  Document  are  used  in  the  AoA  to  quantify 
points  of  differentiation. 
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Simulations  help  address  uncertainty  across  likely  operational  contingencies  (Ford  & 
Dillard,  2008;  2009,  May  13-14;  2009,  July).  Balancing  of  programmatic  and  operational 
risks  should  be  accounted  for,  but  costs  predominate  most  analyses  because,  for  major 
weapon  systems,  they  can  be  huge.  Not  only  must  research  and  development  costs  be 
considered,  but  also  the  production  costs,  and,  beyond  that,  all  of  the  operating  costs  of 
spares,  diagnostics,  maintenance,  tools,  training  manuals,  etc.,  must  be  estimated. 

While  the  emphasis  is  clearly  on  cost  in  these  stages,  operational  effectiveness  must 
also  be  considered  because  that  is  where  benefits  are  realized.  Current  guidance  does  not 
provide  a  method  for  estimating  benefits  in  common  units.  Some  feel  that  the  emphasis  is 
disproportionately  on  cost  without  enough  emphasis  on  benefits.  In  current  guidance  (e.g., 
the  GAO’s  Cost  Estimation  Guide,  2009,  p.  35)  benefits  are  primarily  in  the  form  of  cost 
avoidance  or  cost  savings.  Clearly,  these  are  not  equivalent  to  normal  estimates  of  benefits 
in  the  business  world  where  revenue  is  the  primary  indicator  of  benefit  and  is  not  derived 
from  the  denominator  or  cost  side  of  the  equation.  Monetizing  benefits  as  some  form  of  cost 
savings/avoidance  leads  to  a  slippery  slope  where  the  only  indicator  of  value,  or  numerator 
in  a  productivity  equation,  such  as  return  on  investment  (ROI),  is  a  derivative  of  the 
denominator,  i.e.,  cost.  Such  predilections  inevitably  lead  to  the  lowest  cost  alternatives, 
which  may  not  provide  the  highest  benefits. 

Research  Focus 

Benefits,  in  common  units,  should  be  included  in  AoA  to  enable  higher  fidelity 
comparisons  among  alternatives  on  the  basis  of  value  and  not  just  cost.  But  how  can 
sponsors  and  program  managers  best  valuate  very  real  and  important  but  intangible 
benefits  such  as  combat  effectiveness,  survivability,  or  national  security?  Lacking  a  credible 
ability  to  quantify  such  subjective  or  intangible  benefits  of  the  capabilities  of  a  system  type 
(or  technical  alternative)  is  a  serious  omission  in  any  rigorous  analysis  of  alternatives.  The 
third  author’s  experience  includes  several  recent  examples  that  illustrate  the  need  for  more 
than  a  conventional  cost  effectiveness  analysis  to  defend  a  program  requirement,  or  a 
system  parameter  of  technical  capability.  Often,  a  particular  system  parameter  of  capability 
(e.g.,  weight,  C-130  transportability,  vertical  take-off  and  landing)  become  a  metric  of 
program  life  and  death,  but  with  notably  sparse  articulation  of  empirical  benefit  to  the 
customer/end  user. 

The  Case  of  the  Javelin  Anti-Tank  Weapon  System 

The  Javelin  anti-tank  weapon  system  was,  when  it  was  conceptualized,  merely 
named  after  its  requirement  as  the  Advanced  Anti-Armor  Weapon  System-Medium  (AAWS- 
M).  In  1987-89, the  US  Army  tested  three  competing  technologies  to  fulfill  the  operational 
need  fora  one-man-portable  anti-armor  weapon  system  in  the  medium  range  (1,000-2,000 
meter)  category,  and  replace  its  aging  and  ineffective  DRAGON  weapon  system.  Principally, 
the  weapon  was  to  do  the  following:  be  able  to  defeat  current  and  projected  threat  armored 
vehicles  (including  tanks),  have  a  maximum  range  of  at  least  2,000  meters,  weigh  no  more 
than  20.5  kg  (with  under  15.5  kg  being  desirable),  have  the  ability  to  be  fired  from  enclosed 
spaces,  and  be  able  to  engage  armored  vehicles  under  cover  or  in  hull  defilade.  The  US 
Marine  Corps  agreed  to  these  requirements,  promising  to  pay  for  production  items,  but  not 
to  fund  research  and  development. 

In  August  of  1986,  “Proof  of  Principle”  contracts  of  $30  million  each  were  awarded  to 
three  competing  contractor  teams,  spanning  a  27-month  period  (the  phase  we  now  call 
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“Technology  Development”)  to  develop  the  technologies  and  conduct  a  “fly-off’  missile 
competition.  Each  offered  the  needed  capability  solutions  with  differing  technologies.  Ford 
Aerospace  teamed  with  its  partner  Loral  Systems,  offering  a  laser  beam-riding  missile. 
Hughes  Aircraft  teamed  with  Boeing  to  offer  a  fiber-optic  guided  missile.  Texas  Instruments 
teamed  with  Martin-Marietta,  offering  an  imaging  infra-red  (I2R)  or  forward  looking  infra-red 
(FUR)  missile  system.  Each  candidate  system  also  offered  some  specific  operational 
advantages  and  disadvantages  that  were  almost  impossible  to  quantify  in  terms  of  cost: 

•  The  Ford/Loral  Laser  Beam  Rider  required  an  exposed  gunner  and  man-in- 
loop  throughout  its  rapid  flight.  It  was  cheapest  at  an  estimated  $90,000  “cost 
per  kill,”  a  figure  comprised  not  only  of  average  production  unit  cost  estimates 
but  also  reliability  and  accuracy  estimates.  It  was  fairly  effective  in  terms  of 
potential  combat  utility,  with  diminishing  probability-of-hit  at  increasing  range. 

•  The  Hughes/Boeing  Fiber-Optic  guided  prototype  enabled  an  unexposed 
gunner  (once  launched)  and  also  required  man-in-loop  throughout  its  slower 
flight.  It  was  costlier,  but  less  affected  by  range  accuracy  with  its  automatic 
lock-on  and  guidance  in  its  terminal  stage  of  flight,  and  even  offered  target 
switching.  It  was  also  more  gunner  training  (learning)  intensive,  but  could 
attack  targets  from  above  where  their  armor  was  thinnest. 

•  The  FLIR  prototype  offered  completely  autonomous  “fire  and  forget”  flight  to 
target  after  launch,  was  perceived  as  both  costliest  and  technologically 
riskiest.  It  would  be  easiest  to  train  and  would  be  effective  to  maximum 
ranges  by  means  of  its  target  acquisition  sensor  and  guidance  packages.  It 
was  an  outgrowth  of  a  1980  initiative  by  the  Defense  Advanced  Research 
Project  Agency  (DARPA)  called  Tank  Breaker  that  also  used  “top  attack”  as  a 
more  effective  means  of  armored  target  defeat. 

1988  was  a  busy  year  for  the  AAWS-M  industry  contractors,  as  well  as  the 
government  acquirers  and  program  sponsors.  All  three  candidate  teams  engaged  with  finally 
building  and  flight-testing  their  missile  prototypes.  They  were  also  submitting  their  bids  to  the 
government’s  Request  for  Proposal  for  the  upcoming  advanced  development  phase.  On  the 
government  side,  acquirers  were  evaluating  these  bids  and  preparing  to  award  the  36- 
month  Engineering  and  Manufacturing  Development  (EMD)  phase  contract,  while  sponsors 
were  completing  a  Cost  and  Operational  Effectiveness  Analysis  (COEA)  of  the  three 
candidate  AAWS-M  materiel  solutions.  Each  of  the  teams  enjoyed  generally  successful 
missile  flight  test  outcomes  as  the  “proof  of  principle”  phase  ended.  Each  flew  over  a  dozen 
missiles  and  achieved  a  target  hit  rate  of  over  60%. 

The  Laser  Beam  Rider  candidate  emerged  the  winner  of  the  COEA,  presumably 
from  weighted  cost/efficiency  factors.  But  in  a  strange  twist,  the  concurrent  deliberation  of 
the  Source  Selection  Evaluation  Board  (SSEB)  instead  chose  the  FLIR  candidate, 
presumably  because  of  a  bias  toward  “fire  &  forget.”  As  part  of  a  typical  capability 
formulation  process,  technical  constraints  are  deliberately  avoided  in  requirements 
documents,  to  allow  and  encourage  a  maximum  range  of  alternative  solutions  to  the  need  or 
capability  deficiency.  While  time  of  flight  and  gunner  survivability  were  not  stated 
requirements  in  the  AAWS-M  Joint  Required  Operational  Capability  document  per  se,  “fire 
and  forget”  nevertheless  translated  into  greatly  enhanced  gunner  survivability,  and 
overwhelmingly  appealed  to  user  representatives  (and  government  developers). 

The  EMD  contract  was  awarded  in  June  of  1989  to  the  Joint  Venture  team  of  Texas 
Instruments  and  Martin-Marietta.  However,  at  about  18  months  into  this  program,  serious 
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technical  problems  doubled  the  expected  cost  of  development  and  added  about  eighteen 
more  months  to  the  originally  planned  thirty-six  months  to  complete.  This  constituted  a 
Nunn-McCurdy  breach  of  cost  and  schedule  thresholds,  with  requisite  Congressional 
notifications  and  formal  re-baselining  taking  the  better  part  of  the  next  year  to  accomplish. 
Various  technical  issues  plagued  the  program  at  this  point,  with  system  weight  being 
perhaps  chief  among  them.  User  representatives  convened  a  Joint  Requirements  Overview 
Council  (JROC)  to  re-evaluate  the  maximum  weight  requirement  of  45  pounds  and 
increased  the  program  threshold  to  49.5  pounds.  Clearly,  the  Army  and  Marine  Corps 
communities  wanted  the  emergent  system  and  its  planned  capabilities.  But  that  didn’t 
resolve  all  of  AAWS-M’s  issues. 

During  the  months  that  the  program  teetered  on  the  brink  of  termination  for  its 
technical  and  business  issues,  the  Director  of  the  OSD  Office  of  Program  Analysis  and 
Evaluation  (DPAE),  as  a  principle  member  of  the  DAB,  took  the  program  to  task  stating  that 
if  the  FUR  version  could  not  be  shown  able  to  achieve  the  same  $90K  “cost  per  kill”  as  had 
been  estimated  for  the  laser  beam-rider,  then  the  program  should  be  terminated  and  re¬ 
started,  changing  technologies  and  pursuing  the  less  risky  laser-guided  version.  The 
principal  cost  driver  of  the  FUR  technology  that  enabled  fire-and-forget  was  a  64x64  matrix 
(of  heat  detectors/pixels)  focal  plane  array  (FPA),  to  be  manufactured  by  one  of  the  Joint 
Venture  partners.  These  tiny  micro-chips  would  comprise  almost  14%  of  the  estimated 
average  unit  production  cost  (UPC)  of  the  entire  missile.  The  ability  of  one  of  the  few 
producers  in  the  world  to  produce  them  with  economically  sufficient  yield,  and  to  achieve 
their  rigorous  performance  specifications  for  sensitivity  seemed,  for  a  while,  to  hold  the  fate 
of  the  entire  program.  Intense  scrutiny  of  projected  yields  and  production  costs  of  these 
critical  components  would  determine  whether  the  program  was  feasible  from  this  aspect 
alone,  some  believed.  But  the  answer  was  somewhat  ambiguous,  with  roughly  $  1 2k  being 
the  target  for  average  UPC,  given  a  planned  buy  quantity  of  about  70,000.  And  cost  of  FPAs 
wasn’t  the  only  problem  with  them.  But  it  turned  out  that  their  benefits  could  be  described  in 
a  fairly  tangible  way. 

The  AAWS-M  FPA  specifications  were  derived  from  a  scenario-based  target  list  of 
potential  threat  vehicles  in  different  environments  of  atmospheric  temperature,  humidity, 
obscuration,  etc.  When  the  user  community  saw  that  early  developmental  AAWS-M  focal 
plane  arrays  were  not  meeting  the  full  specifications,  they  convened  another  JROC  to  allow 
stepped,  incremental  achievement  of  target  defeat  scenarios  over  time — something  we 
would  now  refer  to  as  evolutionary  growth.  They  stratified  performance  in  terms  of  levels  A, 

B  and  C  to  convey  degrees  of  target  defeat  capability  in  FPAs — a  very  unusual  move  by 
sponsors,  having  to  dissect  a  requirement  to  accommodate  the  pace  of  technological 
achievement.1  This  provided  a  qualitative  assessment  of  what  was  achievable  and 
satisfactory  for  system  performance.  Once  again,  the  communities  that  needed  AAWS-M’s 
capabilities  were  trying  to  ease  the  path  forward. 

Fortunately,  independent  program  evaluation  teams  also  reported  that  FUR 
technology  was  progressing  and  would  be  achievable  within  a  re-baselined  program.  This 
joint  position,  along  with  wider  program  advocacy,  curtailed  the  technical  and  business 
arguments  and  the  fire-and-forget  Javelin  was  allowed  to  proceed.  An  additional  and  more 
capable  provider  of  FPAs  was  brought  in  and  accelerated  as  a  second  source  for  this  critical 


1  Perhaps  not  unlike  today’s  emergence  of  an  Apple  iPhone®  being  followed  soon  after  by  release  of  a  3G- 
capable  iPhone®. 
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component.  After  still  more  and  difficult  advanced  development  program  challenges,  AAWS- 
M  eventually  became  the  Javelin — and  is  known  today  as  one  of  our  most  successful 
combat  systems.  (In  the  end,  soldiers  and  Marines  never  had  to  accept  to  accept  B  and  C- 
level  FPA  performance,  as  the  full-capable  FPA  technology  did  in  fact  emerge  in  time  for 
fielding.  And  system  weight  has  been  held  just  below  49.5  pounds  throughout  its  many 
years  of  production.) 

There  are  many  business  and  public  policy  lessons  to  be  learned  from  the  Javelin 
program.  Within  its  long  saga  from  initial  concept  to  modern-day  deployment  and  combat 
use  are  illustrations  of  requirements  capture,  early  prototyping,  technology  readiness, 
modeling  and  simulation,  economic  forces  of  competition,  acquisition  strategy,  decision 
bureaucracy,  product  discovery,  and  economies  of  scale,  etc.  Perhaps  the  best  lesson 
learned  from  the  case  presented  here  about  analyzing  alternatives  is  that  a  single,  unstated, 
qualitative  factor  of  performance  (gunner  survivability)  ultimately  drove  the  choice.  Javelin 
had  a  requirements  document  with  many  pages  of  quantifiable  requirements  stated  as 
measures  of  performance  and  effectiveness.  But  the  parameter  of  system  technology  that 
promised  the  most  of  what  was  impossible  to  quantify  became  the  overriding  factor  in  the 
selection  of  alternatives.  A  magazine  advertisement  purchased  by  the  Joint  Venture  shortly 
after  their  EMD  contract  win  said  it  eloquently:  “Fire  &  Forget  AAWS-M:  The  Gunner  Wins.” 
The  failure  of  the  Javelin  program  to  move  to  the  final  solution  faster  and  more  directly  is 
due  in  large  part  to  the  insufficient  articulation  of  benefits  as  part  of  the  Analysis  of 
Alternatives  process. 

Research  Question 

As  illustrated  by  the  Javelin  program,  there  is  a  basic  need  for  the  use  of  a  common 
units  of  benefit  estimate  in  the  Analysis  of  Alternatives  process.  This  should  lead  to  including 
common  units  benefits  estimates  as  well  as  costs  in  the  acquisition  AoAs.  The  problem  is  to 
develop  a  means  to  do  this  more  effectively,  given  the  nebulous  nature  of  so  many  of  the 
critical  benefits  of  weapon  systems.  How  can  such  a  method  be  consistently  applied  to 
many  alternatives  across  a  wide  range  of  operational  conditions?  The  current  research 
examines  how  KVA  can  be  integrated  with  system  dynamics  modeling  to  generate 
defensible  common  units  of  benefit  estimates  that  will  improve  the  rigor  of  the  AoA 
process  and  thereby  improve  acquisition  processes. 

The  goals  of  the  current  work  are: 

•  Examine  how  military  operations  systems  dynamics  simulations  can  be 
combined  with  the  KVA  approach, 

•  Identify  potential  advantages  and  disadvantages  of  integrating  military 
operations  simulations  and  the  KVA  approach, 

•  Investigate  the  potential  of  exploiting  the  benefits  from  the  synergy  of  SD  and 
KVA  to  improve  acquisition  AoA  processes,  and 

•  Identify  and  describe  potential  implications  of  the  integration  on  acquisition 
practice. 

Due  to  the  preliminary  nature  of  this  proof-of-concept  study,  precise  descriptions  of 
system  operations  are  necessary.  The  focus  is  on  the  potential  usefulness  of  integrating  SD 
and  KVA. 
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Introduction  to  Knowledge  Value  Analysis 

In  the  US  Military  context,  the  Knowledge  Value  Added  (KVA)  methodology  is  a  new 
way  of  approaching  the  problems  of  estimating  the  productivity  (e.g.,  in  terms  of  ROI)  for 
military  capabilities  embedded  in  processes  such  as  the  CONOPS  for  a  weapons  system.  In 
the  current  study,  we  posited  several  alternative  CONOPS  for  a  UAV  system  and  used 
system  dynamic  modeling  to  evaluate  their  relative  productivity.  The  KVA  approach  was 
used  to  estimate  the  parameters  based  on  the  system  dynamic  models  by  providing  the 
estimates  of  the  relative  productivity  (i.e. ,  the  ROI2)  of  each  alternative. 

In  a  broader  context,  KVA  also  addresses  the  requirements  of  the  many  Department 
of  Defense  (DoD)  policies  and  directives  previously  reviewed  by  providing  a  means  to 
generate  comparable  value  or  benefit  estimates  for  various  processes  and  the  technologies 
and  people  that  execute  them.  It  does  this  by  providing  a  common  and  relatively  objective 
means  to  estimate  the  value  of  new  technologies  as  required  in  the: 

•  Clinger-Cohen  Act  of  1996  that  mandates  the  assessment  of  the  cost  benefits 
for  information  technology  investments. 

•  Government  Accountability  Office’s  Assessing  Risks  and  Returns:  A  Guide 
for  Evaluating  Federal  Agencies’  IT  Investment  Decision-Making,  Version  1 , 
(February  1997)  that  requires  that  IT  investments  apply  ROI  measures. 

•  DoD  Directive  8115.01 ,  issued  October  2005,  that  mandates  the  use  of 
performance  metrics  based  on  outputs,  with  ROI  analysis  required  for  all 
current  and  planned  IT  investments. 

•  DoD  Risk  Management  Guidance  Defense  Acquisition  guide  book  that 
requires  alternatives  to  the  traditional  cost  estimation  be  considered  because 
legacy  cost  models  tend  not  to  adequately  address  costs  associated  with 
information  systems  or  the  risks  associated  with  them. 

KVA  is  a  methodology  that  describes  all  organizational  outputs  in  common  units. 

This  provides  a  means  to  compare  the  outputs  of  all  assets  (human,  machine,  information 
technology)  regardless  of  the  aggregated  outputs  produced.  Thus,  it  provides  insights  about 
the  productivity  level  of  processes,  people,  and  systems  in  terms  of  a  ratio  of  common  units 
of  output  produced  by  each  asset  (a  measure  of  benefits)  divided  by  the  cost  to  produce  the 
output  By  capturing  the  value  of  knowledge  embedded  in  an  organization’s  core 
processes,  employees  and  technology,  KVA  identifies  the  actual  cost  and  value  of  a  people, 
systems,  or  processes.  Because  KVA  identifies  every  process  required  to  produce  an 
output  and  the  historical  costs  of  those  processes,  unit  costs  and  unit  values  of  outputs, 
processes,  functions  or  services  are  calculated.  An  output  is  defined  as  the  end-result  of  an 
organization’s  operations;  it  can  be  a  product  or  service,  as  shown  in  Figure  1. 


2  ROI  is  defined  as  the  revenue-cost/cost  where  revenue  is  defined  as  the 
a  market  comparables  approach.  Given  that  the  price  per  common  unit  is 
arable  price,  i.e.,  revenue,  is  not  required. 
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For  the  purpose  of  the  systems  dynamics  model  developed  for  this  study,  KVA  was 
used  to  describe  the  outputs  of  all  the  processes  and  subprocesses  in  common  units.  This 
allowed  us  to  make  their  relative  performance  (e.g.,  productivity,  ROIs)  comparable.  KVA 
was  used  to  measure  the  value  added  by  the  human  capital  assets  (i.e.,  military  personnel 
executing  the  processes)  and  the  system  assets  by  analyzing  the  processes  performances. 
KVA  provided  a  means  to  set  the  systems  dynamic  model  parameters  so  that  the  results 
would  provide  a  means  to  compare  the  performance  of  various  approaches  to  the  system 
problem. 

By  capturing  the  value  of  knowledge  embedded  in  systems  and  in  use  in  operators  of 
the  processes,  KVA  identified  the  productivity  of  the  system-process  alternatives.  Because 
KVA  identified  every  process  output  required  to  produce  the  final  aggregated  output,  the 
common  unit  costs  and  the  common  unit  values  were  estimated.  This  allowed  for  the 
benchmarking  of  various  systems  and  the  processes  they  support  with  any  other  similar 
processes  across  the  military. 

The  KVA  methodology  has  been  applied  in  over  80  projects  within  the  DoD,  from 
flight  scheduling  applications  to  ship  maintenance  and  modernization  processes  to  the 
current  project  analyzing  several  alternative  approaches  to  the  system  alternatives  problem. 
In  general,  the  KVA  methodology  was  used  for  this  study  because  it  could: 

•  Compare  alternative  approaches  modeled  with  a  systems  dynamics  model  in 
terms  of  their  relative  productivity, 

•  Allocate  value  and  costs  to  common  units  of  output, 

•  Measure  value  added  by  the  system  alternatives  based  on  the  outputs  each 
produced,  and 

•  Relate  outputs  to  cost  of  producing  those  outputs  in  common  units. 

Describing  processes  in  common  units  also  permits,  but  does  not  require,  market 
comparable  data  to  be  generated,  particularly  important  for  non-profits  like  the  US  Military. 
Using  a  market  comparables  approach,  data  from  the  commercial  sector  can  be  used  to 
estimate  price  per  common  unit,  allowing  for  revenue  estimates  of  process  outputs  for  non¬ 
profits.  This  also  provides  a  common  units  basis  to  define  benefit  streams,  regardless  of  the 
process  analyzed. 
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KVA  differs  from  other  nonprofit  ROI  models  because  it  can  allow  for  revenue 
estimates,  enabling  the  use  of  traditional  accounting,  financial  performance,  and  profitability 
measures  at  the  sub-organizational  level.  KVA  can  rank  processes  or  process  alternatives 
by  their  relative  ROIs.  This  assists  decision-makers  in  identifying  how  much  various 
processes  or  process  alternatives  add  value. 

In  KVA,  value  is  quantified  in  two  key  metrics:  Return-on-Knowledge  (ROK: 
revenue/cost)  and  ROI  (revenue-investment  cost/investment  cost).  The  raw  data  from  a  KVA 
analysis  can  become  the  input  into  the  ROI  models  and  various  forecasting  techniques  such 
as  real  options  analysis,  portfolio  optimization,  Monte  Carlo  simulation.  By  tracking  the 
historical  volatility  of  price  and  cost  per  unit  as  well  as  ROI,  it  is  possible  to  establish  risk  (as 
compared  to  uncertainty)  distributions,  which  is  important  for  accurately  estimating  the 
forecasted  values  for  portfolio  optimization  and  real  options  analysis. 

Introduction  to  System  Dynamics 

The  system  dynamics  methodology  applies  a  control  theory  perspective  to  the 
design  and  management  of  complex  human  systems.  System  dynamics  combines  servo¬ 
mechanism  thinking  with  computer  simulation  to  analyze  systems.  It  is  one  of  several 
established  and  successful  approaches  to  systems  analysis  and  design  (Flood  &  Jackson, 
1991;  Lane  &  Jackson,  1995;  Jackson,  2003).  Forrester  (1961)  develops  the  methodology's 
philosophy,  and  Sterman  (2000)  specifies  the  modeling  process  with  examples  and 
describes  numerous  applications.  The  methodology  has  been  extensively  used  for  this 
purpose,  including  studying  development  projects.  The  system  dynamics  perspective 
focuses  on  how  the  internal  structure  of  a  system  impacts  system  and  managerial  behavior 
and,  thereby,  performance  over  time.  The  approach  is  unique  in  its  integrated  use  of  stocks 
and  flows,  causal  feedback,  and  time  delays  to  model  and  explain  processes,  resources, 
information,  and  management  policies.  Stocks  represent  accumulations  or  backlogs  of  work, 
people,  information,  or  other  portions  of  the  system  that  change  overtime.  Flows  represent 
the  movement  of  those  commodities  into,  between,  and  out  of  stocks.  The  methodology’s 
ability  to  model  many  diverse  system  components  (e.g.,  work,  people,  money,  value), 
processes  (e.g.,  design,  technology  development,  production,  operations,  quality 
assurance),  and  managerial  decision-making  and  actions  (e.g.,  forecasting,  resource 
allocation)  makes  system  dynamics  useful  for  modeling  and  investigating  military  operations, 
the  design  of  materiel,  and  acquisition. 

When  applied  to  acquisition  programs,  system  dynamics  has  focused  on  how 
performance  evolves  in  response  to  interactions  among  development  strategy  (e.g., 
evolutionary  development  versus  traditional),  managerial  decision-making  (e.g.,  scope 
developed  in  specific  blocks),  and  development  processes  (e.g.,  concurrence).  System 
dynamics  is  appropriate  for  modeling  acquisition  because  of  its  ability  to  explicitly  model 
critical  aspects  of  development  projects.  System  dynamics  models  of  development  projects 
are  purposefully  simple  relative  to  actual  practice  to  expose  the  relationships  between 
causal  structures  and  the  behavior  and  performance  that  they  create.  Therefore,  although 
many  processes  and  features  of  system  design  and  participants  interact  to  determine 
performance,  only  those  that  describe  features  related  to  the  topic  of  study  are  included. 

The  importance  of  deleted  features  can  be  tested  when  system  dynamics  is  used  to  test  the 
ability  of  the  model  structure  to  explain  system  behavior  and  performance. 
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Based  on  the  preceding  and  the  authors’  experience  with  system  dynamics,  there 
appears  to  be  an  opportunity  to  exploit  the  capabilities  of  the  system  dynamics  methodology 
to  make  the  Knowledge  Value  Added  approach  more  accurate. 

Research  Methodology 

In  the  current  work,  Knowledge  Value  Added  and  system  dynamics  were  integrated 
to  test  their  ability  to  improve  the  precision  of  AoAs  in  acquisition  programs.  A  generic 
structure  of  a  mobile  weapon  system  process  was  first  developed  and  tested  using  the 
system  dynamics  methodology.  Then,  KVA  value  and  cost  estimates  were  operationalized 
in  the  system  dynamics  model.  The  model  was  calibrated  to  reflect  four  extant  weaponized 
Unmanned  Aerial  Vehicles  (UAVs).  One  of  those  calibrations  was  used  as  the  basis  for 
using  the  model  in  a  hypothetical  AoA  for  upgrading  the  UAV  to  address  a  different  type  of 
target.  Simulation  results  were  analyzed  to  test  the  ability  of  the  system  dynamics  model  to 
estimate  benefits  streams  using  KVA  in  terms  of  the  relative  value  added  of  the  capabilities 
of  the  system. 

A  Generic  Model  of  Mobile  Weapons  Use 

The  model  has  three  sectors:  weapons  movement,  target  evolution,  and  KVA 
analysis.  As  will  be  described,  the  model  structure  simulates  two  critical  aspects  of  mobile 
weapon  system  operations:  1)  the  support  and  movement  of  the  weapon  and  2)  target 
evolution  from  identification  through  confirmation  of  destruction. 

The  Weapons  Movement  Sector 

The  Weapons  Movement  sector  of  the  model  simulates  the  positions  and 
movements  of  weapons  (e.g.,  individual  UAVs  or  Javelin  gunners).  Figure  2  shows  the 
positions  that  weapons  (generically  called  “assets”)  can  take  (boxes)  and  the  rates  of  their 
movements  from  one  position  to  another  (arrows  between  boxes).  It  is  assumed  that  the 
total  number  of  assets  remains  constant,  i.e.,  no  weapons  are  added  or  lost  during 
operations.  This  assumption  can  be  relaxed  when  modeling  a  specific  asset.  The  movement 
of  weapons  is  a  subprocess  of  operating  the  weapon  system  that  adds  value  and  imbeds 
learning  into  tools,  requires  learning  time  for  operators  to  be  capable  of  doing,  and  requires 
processing  time  to  accomplish.  Therefore,  the  completion  of  moving  weapons  to  the  station 
and  back  to  the  base  is  an  output  of  that  subprocess  and  an  input  to  the  KVA  analysis.  The 
combination  of  the  two  movements  “Assets  arrive  at  station  rate”  and  “Assets  arriving  as 
base  rate”  represent  the  accomplishment  of  the  vehicle  movement  subprocess. 
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•Avg  time  at  base 


Figure  1.  Positions  and  Movement  of  Weapons  during  Operations 

Each  rate  in  the  weapons  sector  describes  the  average  movement  of  the  weapons  in  the 
accumulation  that  precedes  the  rate.  Each  rate  is  defined  with  the  number  of  weapons 
preparing  for  that  rate  (to  leave  base  or  station)  or  event  (to  arrive  at  station  or  base)  and 
the  average  time  spent  by  a  weapon  in  the  preceding  accumulation.  For  example,  the 
(average)  “Assets  leaving  base  for  station”  rate  is  equal  to  the  number  of  assets  at  the  base 
divided  by  the  average  time  that  a  weapon  spends  at  the  base  between  trips  to  the  station. 
This  formulation  increases  the  average  departure  rate  with  more  weapons  at  the  base  and 
decreases  the  average  departure  rate  if  weapons  stay  at  the  base  longer.  The  average  time 
at  the  base  is  characteristic  of  particular  assets  and  can  generate  different  behaviors  and 
performances  across  weapons  and  configurations. 

The  Target  Evolution  Sector 

The  target  evolution  sector  of  the  model  simulates  the  development  of  targets 
through  five  subprocesses  of  system  operations. 

1.  Acquire  target:  Includes  detection,  recognition,  location,  classification 
(identification),  and  confirmation  (Lombardo,  2003;  Global  Security,  2010). 

2.  Fire  support  coordination:  Allocates  targets  to  weapons  by  a  group  of 
people  that  have  access  to  information  about  the  battlefield  situation,  and 
doctrine,  major  systems,  significant  capabilities  and  limitations  and  often  their 
TTP  [tactics,  techniques  and  procedures]  (Williams,  2001). 

3.  Fire  mission  development:  Prepares  specific  instructions  and  target 
information  for  transmission  to  the  weapons  team  and  to  the  weapon  (e.g., 
target  location  coordinates). 

4.  Engage  target:  Weapons  operators  (e.g.,  pilots  for  UAV)  maneuver  the 
weapon  within  striking  distance  of  the  target,  enter  the  target  coordinates  and 
launch  munitions  . 

5.  Battlefield  assessment:  Often  the  same  asset  as  was  used  for  target 
acquisition  is  used  to  evaluate  the  success  of  engagement  in  destroying  the 
target. 
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In  the  model,  targets  evolve  through  these  stages  in  an  “aging  chain”  structure  of 
sequential  accumulations  (backlogs  +  work  in  progress,  referred  to  here  as  backlogs)  and 
(sub)processes  that  drain  those  backlogs  and  contribute  to  the  backlog  of  the  next 
downstream  subprocess.  Figure  3  shows  the  conditions  of  targets  (boxes)  and  the  rates  of 
their  movements  from  one  condition  to  another  (arrows  between  boxes)  due  to 
subprocesses.  The  movements  “Acquire  target  completion  rate,”  “Fire  support  coordination 
to  asset,”  “Fire  mission  completion  rate,”  “Engage  target,”  and  “Battlefield  assessment  rate” 
are  subprocesses  that  add  value,  imbed  learning  in  tools,  require  learning  time  for  operators 
to  be  capable  of  doing,  and  require  processing  time  to  complete.  Therefore,  they  are  each 
outputs  of  those  subprocesses  and  inputs  to  the  KVA  analysis. 


Lose  target  after  miss  rate 


Figure  2.  Accumulations  and  Movements  of  Targets  in  Weapon  System  Operations 

In  addition  to  the  primary  flows  of  targets  through  the  subprocesses,  the  targets 
sector  models  three  common  causes  of  mission  failure:  1)  hitting  the  target  but  failing  to 
destroy  it,  2)  missing  the  target,  and  3)  missing  the  target  and  losing  the  location  information 
needed  to  engage  the  target  again  (e.g.,  because  the  target  moved).  Each  cause  moves  the 
target  to  a  different  condition  in  the  target  aging  chain.  Hitting  the  target  but  failing  to  destroy 
it  (e.g.,  a  hardened  target)  requires  reengagement  but  often  no  additional  targeting 
information.  After  Battlefield  assessment,  these  targets  are  returned  to  the  Target 
engagement  backlog.  Missing  the  target  (e.g.,  a  small  target)  requires  that  the  fire  mission 
be  developed  again  to  re-aim  the  weapon  prior  to  reengagement.  Therefore,  these  targets 
are  returned  to  the  Targets  in  fire  mission  development  backlog  after  Battlefield  assessment. 
Losing  the  target  (e.g.,  a  fast  moving  vehicle)  requires  that  the  target  be  reacquired. 
Therefore,  these  targets  are  returned  to  the  Targets  being  acquired  backlog  after  Battlefield 
assessment. 

In  a  manner  similar  to  the  modeling  of  the  movement  of  weapons,  the  rates  in  the 
targets  sector  describe  the  average  movement  of  targets  between  backlogs.  The  primary 
rates  in  the  aging  chain  are  defined  by  the  number  of  targets  in  the  backlogs  of  the 
subprocess  and  the  average  time  required  to  perform  the  subprocess.  The  average  time 
required  to  perform  the  subprocess  is  characteristic  of  particular  subprocesses  (e.g., 
different  engagement  durations  for  different  weapons)  and  can  generate  different  behavior 
and  performance  across  weapons  and  configurations.  When  two  or  more  flows  drain  a 
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backlog  (Targets  in  fire  support  coordination  and  Battlefield  assessment  backlog)  the  total 
outflow  is  split  between  the  flows  with  a  percent  that  leaves  the  stock  through  each  outflow. 
The  return  flows  are  each  a  fraction  of  the  Battlefield  assessment  rate.  Those  fractions  are 
based  on  the  ability  of  the  weapon  to  successfully  destroy,  hit,  and  not  lose  targets. 
Therefore,  like  in  practice,  different  weapon  alternatives  (e.g.,  range,  payload,  dash  speed) 
impact  mission  success.  The  model  use  section  of  this  report  describes  how  these  features 
of  the  model  were  used  to  describe  operational  scenarios  and  weapon  configurations.  The 
fraction  of  the  targets  that  are  returned  due  to  being  hit  but  not  destroyed  (to  engagement 
backlog),  missed  (to  mission  development  backlog),  or  lost  (to  target  acquisition  backlog) 
battlefield  assessment  rate  are  described  with  the  probability  of  destruction  if  the  target  is  hit 
with  the  ordinance  (p(kill  if  hit))  or  p(kill)),  the  probability  of  the  weapon  hitting  the  target  with 
ordinance  (p(hit)),  and  the  probability  of  not  losing  the  target  if  it  is  missed  with  the 
ordinance  (p(not  lose)),  respectively.3  The  probabilities  are  determined  by  comparing  the 
ability  of  a  weapon  to  successfully  destroy,  hit,  and  not  lose  targets  to  the  characteristics  of 
the  target  and  a  function  that  describes  how  the  weapon’s  ability  compared  to  the  target 
impacts  weapon  performance.  More  specifically,  the  probability  of  kill  is  modeled  with  the 
weapon’s  payload  compared  to  the  lethal  payload  (i.e.,  ordinance  size  required  to  kill); 
probability  of  hit  is  modeled  with  the  weapon’s  dash  speed  compared  to  the  target’s  speed; 
and  the  probability  of  not  losing  the  target  is  modeled  with  the  weapon’s  range  compared  to 
the  target’s  distance  from  the  base.  Therefore: 

p(kill)  =  fk(Payload  /  Lethal  payload) 

p(hit)  =  fh(Dash  speed  /  Target  speed) 

p(not  lose)  =  fni(Range  /  Target  distance  from  base) 

where: 

p(kill)  -  probability  of  destruction  if  the  target  is  hit  with  the  ordinance 
p(hit)  -  probability  of  the  weapon  hitting  the  target  with  ordinance 
p(not  lose)  -  probability  of  not  losing  the  target  if  it  is  missed  with  the 
ordinance 

The  three  functions  that  estimate  the  probabilities  based  on  the  ratios  are  assumed 
to  be  simple  but  realistic  relations  that  include  the  entire  range  of  possible  conditions.4 The 
function  relating  the  Payload/Lethal  payload  ratio  to  the  probability  of  kill  is  assumed  to 
increase  linearly  from  p(kill)=0  when  the  ratio  is  zero  (i.e.,  no  payload  prevents  any  chance 
of  target  destruction)  to  p(kill)=1 00%  when  the  ratio  is  greater  than  or  equal  to  1  (i.e.,  if  the 
payload  exceeds  the  lethal  payload  the  target  is  assumed  to  be  destroyed  if  it  is  hit).  The 
function  relating  the  Dash  speed/Target  speed  ratio  to  the  probability  of  the  weapon  hitting 
the  target  with  ordinance  assumes  that  the  vehicle  will  “chase”  a  moving  target  and  that  the 
faster  the  vehicle  is  then  the  closer  it  can  get  to  the  target  before  releasing  ordinance, 
increasing  the  likelihood  of  hitting  the  target  with  the  ordinance.  However,  there  is  always 
some  possibility  of  missing  a  target  even  if  the  vehicle  is  faster  than  and,  therefore,  close  to 
the  target.  The  function  is  assumed  to  have  an  elongated  “S”  shape  from  p(hit)=0  when  the 
ratio  is  zero  (i.e.,  no  Dash  speed  prevents  hitting  the  target)  to  p(hit)=90%  when  the  ratio  is 
greater  than  or  equal  to  three  (i.e.,  high  likelihood  of  hit  if  weapon  speed  far  exceeds  target 


3  The  probability  of  not  losing  a  target  is  used  instead  of  the  probability  of  losing  a  target  to  retain  a  “bigger  is 
better”  standard  for  all  three  measures  and,  therefore,  facilitate  intuitive  understanding  of  the  model. 

4  These  functions  can  be  described  more  accurately  with  additional  weapon  testing  information. 
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speed.5  The  function  relating  the  Range/Target  distance  from  base  ratio  to  the  probability  of 
not  losing  the  target  if  it  is  missed  with  the  ordinance  assumes  that  the  vehicle  will  move 
toward  the  target  but  that  the  target  may  also  move,  sometimes  closer  to  the  vehicle  and 
sometimes  away  from  it.  When  the  target  moves  away  from  the  vehicle  it  may  move  out  of 
the  vehicle’s  range,  causing  the  vehicle  to  lose  the  target.  The  function  is  assumed  to  have 
a  stretched  out  “S”  shape  from  p(not  lose)=0  when  the  ratio  is  zero  (i.e.,  no  weapon  range 
causes  the  vehicle  to  always  lose  the  target)  to  p(not  lose)=95%  when  the  ratio  is  greater 
than  or  equal  to  1 .8,  reflecting  some  chance  of  losing  the  target  even  if  it  is  well  within  the 
vehicle’s  range. 

The  KVA  Sector 

The  KVA  metrics  were  fully  operationalized  within  the  system  dynamics  model.  The 
KVA  sector  uses  operations  information  from  the  weapons  and  targets  sectors  of  the  model 
and  characteristic  descriptors  of  weapons  to  generate  relative  value  metrics  for  each 
subprocess  (including  weapons  capability  outputs)  of  the  UAV  operations.  KVA  generates  a 
productivity  ratio  that  reflects  output/input.  If  monetized,  this  ratio  can  be  a  traditional 
benefit:cost  ratio  (e.g.,  ROI  if  benefits  are  monetized  as  a  form  of  revenue  surrogate).  Other 
measures  of  benefits  and  costs  can  also  be  used,  as  long  as  there  are  common  units  in  the 
numerator  (benefits)  and  all  the  cost  units  of  all  the  contributors  to  the  denominator  are  the 
same  (as  is  most  often  the  case  because  costs  are  almost  always  monetized),  so  they  can 
each  be  aggregated.  At  each  point  in  time,  each  subprocess’s  productivity  is  the  benefits  it 
has  generated  divided  by  the  costs  to  generate  those  benefits  (i.e.,  output/input). The  model 
includes  both  monetized  and  time-based  KVA  metrics.  However,  it  was  decided  that  the 
current  model  would  be  simpler  to  interpret  by  using  the  non-monetized  common  units  of 
output  (as  described  in  terms  of  the  units  of  time  it  would  take  the  average  person  to  learn 
how  to  produce  the  outputs)  as  the  numerator.  This  results  in  the  output  (i.e.,  common  units 
of  learning  time)/input  (cost  to  produce  the  outputs)  standard  definition  of  productivity. 

A  time-based  application  of  KVA  uses  Learning  Time  to  quantify  benefits  and  Touch 
Time  to  quantify  costs.  Learning  time  captures  the  benefits  derived  from  human  processing 
(e.g.,  flying  the  vehicle),  automated  processes  (e.g.,  take  offs  and  landings),  and 
(importantly)  technologies  integrated  into  the  weapon.  One  of  several  ways  to  quantify  a 
subprocess’s  Learning  Time  is  to  estimate  the  average  time  required  for  a  common  point  of 
reference  learner  to  be  trained  and  become  competent  in  performing  the  subprocess.  Each 
subprocess  is  assigned  a  Unit  Learning  Time  that  reflects  the  relative  (compared  to  other 
subprocesses)  complexity  of  the  subprocess. 

The  denominator  of  KVA  subprocess  productivity  ratios  represents  subprocess 
costs.  Each  subprocess  is  assigned  a  Unit  Touch  Time  that  reflects  the  relative  (compared 
to  other  subprocesses)  effort  required  to  perform  the  subprocess.  The  total  time  spent 
performing  a  subprocess  at  any  given  time  is  the  product  of  the  average  time  required  for 
the  subprocess  and  the  number  of  performances  of  the  subprocess.  Therefore,  a  set  of 
equations  for  estimating  the  costs  of  a  single  subprocess  in  each  time  period  are: 

Subprocess  learning  Time  accumulated  to  date  =  ^  (Rate  of  subprocess  operation  * 


5  The  assumed  function  relating  the  dash  speed/target  speed  to  the  probability  of  hitting  the  target  is  probably 
lower  than  current  experience,  but  is  used  to  reflect  the  change  in  targets  described  in  the  Model  Use  section  of 
this  report. 
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Subprocess  unit  learning  time)  *  dt 

Subprocess  touch  time  accumulated  to  date  =  ^  (Rate  of  subprocess  operation  * 

Subprocess  unit  touch  time  *  dt 

Subprocess  Productivity  =  Subprocess  learning  time  accumulated  to  date  / 

Subprocess  touch  time  accumulated  to  date 

Learning  Times  and  Touch  Times  are  also  aggregated  across  subprocesses  to 
estimate  the  productivity  or  the  entire  operation.  This  allows  the  comparison  of  different 
asset  configurations  (alternatives). 

Model  Calibration  and  Testing 

The  KVA+SD  model  was  calibrated  to  the  operations  of  four  actual  weaponized 
UAVs.  The  operations  included  the  following  subprocesses: 

1.  acquire  target 

2.  fire  support  coordination 

3.  fire  mission  development 

4.  move  weapons 

5.  engage  targets 

6.  battlefield  assessments 

Three  of  the  four  actual  UAVs  are  operational:  Predator,  Sky  Warrior,  and  Reaper. 
The  fourth  UAV  is  the  X-47B.  Basic  characteristics  relating  to  performance  were  collected 
for  the  four  UAVs  from  publicly  available  sources  (e.g.,  Global  Security,  2010).  That 
information  included  vehicle  range,  total  mission  time,  time  on  station,  dash  speed,  and 
payload.  In  some  cases,  multiple  versions  of  the  vehicle  with  different  characteristics  have 
been  developed.  In  these  cases,  a  single  version  was  selected  and  used.  Other  information 
was  estimated  for  each  vehicle’s  operations,  including  learning  and  processing  times  for 
each  subprocess.  Reasonable  assumptions  were  used  in  making  these  estimates,  such  as 
that  the  time  required  to  engage  a  target  after  arriving  on  station  is  inversely  proportional  to 
the  vehicle’s  dash  speed  (i.e.,  faster  dash  speeds  reduce  the  time  required  to  engage). 
These  estimates  are  rough,  but  adequate  for  this  proof-of-concept  study,  which  seeks  to 
determine  if  the  model  is  capable  of  reflecting  differences  in  characteristics  in  KVA 
parameters,  not  predict  actual  outcomes. 

The  model  was  tested  using  standard  tests  for  system  dynamics  models  (Forrester  & 
Senge,  1980;  Sterman,  2000),  including  for  structural  similarity  to  the  actual  system, 
reasonable  behavior  over  a  wide  range  of  input  values,  and  behavior  similarity  to  actual 
systems.  Basing  the  model  on  previously  validated  models,  the  literature  improves  the 
model’s  structural  similarity  to  actual  acquisition  projects,  as  practiced.  Model  behavior  (e.g., 
simulated  sizes  of  backlogs  for  subprocesses  and  rates  of  performing  operations)  were 
compared  to  typical  behavior  and  found  to  be  similar.  For  example,  before  operations  start 
at  the  beginning  of  the  operational  scenario  (described  in  what  follows)  the  backlogs  are 
empty  and  no  operations  are  being  performed.  The  appearance  of  targets  increases 
subprocess  backlogs  and  rates  of  operation  as  weapons  leave  base  and  subsequently 
arrive  on  station,  targets  are  acquired,  fire  coordinated,  missions  developed,  targets 
engaged,  and  the  battlefield  assessed. 
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In  the  evolution  of  targets,  these  backlogs  and  subprocesses  increase  sequentially 
through  the  series  of  operations.  The  growth  of  operations  and  backlogs  slows  as  capacities 
adjust  to  demand  (backlog  sizes),  until  the  operations  are  in  dynamic  equilibrium  conditions 
with  sizes  of  backlogs  and  operations  rates  remaining  within  a  relatively  narrow  range.  This 
represents  “steady  state”  operations  that  could  be  continued  for  a  significant  period  of  time, 
e.g.,  until  damage  to  weapons  or  maintenance  (not  included  in  the  current  model)  change 
weapon  availability.  Model  behavior  was  also  tested  with  extreme  input  values  such  as 
perfect  operations  (e.g.,  probability  of  hit=  1 00%)  and  very  large  versus  very  small  number  of 
weapons  and  targets  as  well  as  more  typical  conditions.  Model  behavior  remained 
defensible  across  wide  ranges  of  input  values,  including  extreme  values.  These  tests 
increase  confidence  that  the  model  generates  realistic  operational  process  behavior 
patterns  due  to  the  same  causal  relations  found  in  the  type  of  operations  investigated  (i.e. , 
generates  “the  right  behavior  for  the  right  reasons”). 

The  operational  scenario  was  described  with  the  quantity  and  characteristics  of  the 
targets.6  A  stream  of  targets  entered  the  target  acquisition  backlog  at  a  steady  rate  of  five 
targets  per  minute.  The  target  distance  from  the  base  was  assumed  to  vary  uniformly  from 
400-1  lOOnm.  This  describes  targets  that  range  from  being  closer  to  the  weapon’s  base  than 
the  shortest  weapon’s  range  to  targets  that  are  farther  from  the  base  than  the  longest 
weapon’s  range.  The  speed  of  the  targets  was  assumed  to  vary  uniformly  from  50  to  250 
nm.  This  describes  targets  that  range  from  those  that  are  immobile  to  targets  that  are  faster 
than  the  fastest  weapon.  The  payload  required  to  destroy  the  target  if  hit  (i.e.,  lethal 
payload)  was  assumed  to  vary  uniformly  from  400  to  1,000  lbs.  This  describes  targets  that 
range  from  being  very  soft  to  very  hardened. 

KVA  productivities  for  the  six  subprocesses  and  the  cumulative  for  those  processes 
for  the  four  Weaponized  UAVs  are  shown  in  Table  1.  For  example,  the  KVA  productivity 
ratios  for  the  Fire  Mission  Development  subprocess  for  the  four  UAV  are  943  (Predator), 
3,122  (Reaper),  1 ,222  (Sky  Warrior),  and  3,962  (X-47B).  They  represent  the  benefits 
(output)  per  unit  of  cost  (input)  and,  therefore,  can  also  be  interpreted  as  a  measure  of  the 
return  on  the  investment,  in  percent.  These  values  remain  constant  in  the  model  after  steady 
state  operations  have  been  established.  As  an  example  of  the  components  of  the  ratios,  the 
Fire  Mission  Development  subprocess  ratio  for  the  Predator  (943)  is  the  quotient  of  the 
accumulated  benefits  (e.g.,  after  5  hours  of  operations)  of  79,684  learning-time  hours  and 
84.5  processing-time  hours.  In  the  simulated  steady  state  operations  this  accumulated 
learning  time  hours  increases  at  a  rate  of  301  learning-time  hours  per  minute  (the  product  of 
the  estimated  500  learning-time  hours  per  fire  development  operation  and  an  average  fire 
development  rate  of  0.6  targets  developed  per  minute)  and  the  processing-time  hours 
increases  at  a  rate  of  0.3  hours  per  minute  (the  product  of  the  estimated  30  minute 
processing  time  to  develop  a  fire  mission  and  the  same  average  fire  development  rate  of  0.6 
targets  developed  per  minute).  Transitional  periods  (e.g.,  start  or  end  of  operations)  or  other 
non-steady  state  operations  can  generate  ratios  that  vary  over  time. 

Table  1.  KVA  Productivity  Ratios 


6  Although  a  single  operational  environment  was  simulated  for  this  research,  multiple  and  different 
environments  can  be  simulated.  Examples  of  characteristics  of  the  operational  scenario  that  can  be  elaborated  to 
include  dynamic  variation  in  the  entering  target  rate,  distributions  of  target  characteristics,  and  more  target 
characteristics. 
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907 
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Note  that,  as  described  above,  these  productivities  are  ratios  of  accumulated 
learning  time  divided  by  accumulated  processing  time.  Therefore,  they  are  relative  values. 

As  expected,  the  three  productivities  for  the  subprocesses  that  are  not  impacted  by  the 
characteristics  of  the  vehicle  (Acquire  targets  Fire  support  coordination,  and  Battlefield 
assessment)  do  not  change.  These  subprocesses  are  not  impacted  by  different  vehicles 
because  the  subprocess  is  the  same  for  all  of  these  vehicles.  The  application  of  system 
dynamics  and  KVA  to  the  Analysis  of  Alternatives  of  other  system  alternatives  such  as 
improved  logistics  or  vehicle  technology  used  for  recognizing  and  indentifying  targets  would 
generate  changes  in  these  KVA  productivities.  However,  three  important  subprocesses  that 
do  impact  total  product  productivity  (“Weapons”  row  in  Table  1)  do  vary  (Fire  mission 
development,  Move  weapons,  and  Engage  targets). 

Some  of  the  ratios  in  Table  1  are  relatively  large  when  compared  to  returns  on 
investment  experienced  in  many  industries,  especially  for  the  engage  targets  subprocess.  A 
primary  reason  is  that  the  numerator  of  these  ratios  includes  the  benefits  of  the  technologies 
incorporated  into  the  UAV  for  target  engagement  purposes.  These  technologies  are 
extremely  complex,  are  reflected  in  very  large  learning-time  hours  that  are  accumulated 
each  time  a  target  is  engaged,  and,  therefore,  generate  high  productivity  ratios.  Similarly, 
the  denominator  of  these  ratios  reflects  the  time  required  to  perform  the  subprocess,  e.g., 
engage  a  target  after  it  has  been  acquired,  fire  support  coordinated,  fire  mission  developed, 
and  the  UAV  moved  to  station.  Actual  engagement  times  are  relatively  short  for  these  UAV, 
further  increasing  the  KVA  productivity  ratios  for  the  engage  target  subprocess.  Differences 
in  learning-times  across  the  UAV  reflect  their  relative  performance  (e.g.,  automation  of 
subprocesses  previously  performed  by  humans)  and  technologies  are  the  primary  causes  of 
differences  in  the  ratios  across  UAV  in  Table  1 .  Therefore,  it  is  reasonable  that  the  very 
large  benefits  of  the  under-development  X-47B  with  its  extremely  advanced  technology 
generate  the  largest  ratios.  Improved  estimates  of  learning-times  and  processing  times  can 
improve  the  accuracy  of  these  ratios.  However,  comparing  the  KVA  productivity  ratios  for  the 
engage  targets  subprocess  with  the  ratios  for  the  move  weapons  subprocess  that  is  simpler 
(lower  numerator)  and  takes  longer  (larger  denominator)  indicates  that  the  rank  order  of  the 
ratios  reflects  the  relative  returns  of  the  different  subprocesses. 

Based  on  these  and  additional  tests,  the  model  is  considered  useful  for  the 
investigation  of  the  integration  of  system  dynamics  and  KVA. 


ACQUISITION  RESEARCH  PROGRAM 

GRADUATE  SCHOOL  OF  BUSINESS  &  PUBLIC  POLICY 

NAVAL  POSTGRADUATE  SCHOOL 


731 


Using  System  Dynamics  and  KVA  to  Improve  AoA 

Consider  the  following  hypothetical  example  of  the  use  of  an  integrated  system 
dynamics/KVA  model  to  improve  the  productivity  estimates  supporting  an  Analysis  of 
Alternatives.  Assume  that  a  new  version  of  the  Predator  UAV  is  being  developed  to  enable  it 
to  engage  opposing  UAVs.  Due  to  the  much  higher  speeds  and  agility  of  UAVs  compared  to 
most  land-based  targets,  the  fraction  of  targets  missed  is  expected  to  be  higher  than  that 
currently  experienced  with  the  Predator.  The  acquisition  program  management  team  has 
access  to  some,  but  limited,  resources  (e.g.,  money,  expert  developer  time  until  required 
delivery,  technology  development  capabilities,  approvals)  to  improve  performance.  Different 
stakeholders  value  payload,  dash  speed,  and  range  differently  and  want  the  program 
management  to  recommend  different  improvements.  Therefore,  program  management 
expects  a  rigorous  review  of  its  Analysis  of  Alternatives  process  and  the  results  that  will 
recommend  one  (and  only  one)  of  the  improvements.  As  part  of  justification  of  the  AoA 
decision,  stakeholders  of  the  two  solutions  not  recommended  are  certain  to  require 
explanations  of  how  and  how  much  the  recommended  improvement  impacts  operational 
performance  compared  to  the  improvements  that  were  not  recommended.  Cost  would,  most 
likely,  be  their  primary  economic  consideration  as  evidenced  by  the  earlier  case  study 
examples.  However,  our  analysis  will  focus  on  value  compared  to  cost  in  terms  of  the 
capabilities  of  the  systems. 

Many  alternatives  have  been  proposed  and  are  being  considered.  A  few  examples 
are7  as  follows: 

•  Increase  the  size  of  the  power  plant,  which  can  be  used  to  increase  the 
vehicle’s  payload,  dash  speed,  or  a  combination  of  both.  This  requires  an 
increase  in  fuel  capacity  to  not  reduce  range. 

•  Redesign  the  transmission,  which  will  increase  the  vehicle’s  dash  speed. 

•  Increase  the  fuel  tank  size,  which  will  increase  the  vehicle’s  range  but 
decrease  its  dash  speed  unless  the  power  plant  is  also  increased. 

•  Reduce  the  time  required  at  base  between  trips  to  station,  which  increases 
the  time  that  the  vehicle  is  on  station  and  available  for  missions. 

Performing  detailed  analysis  of  all  the  possible  alternatives,  such  as  by  building  and 
testing  prototypes  or  very  detailed  simulations,  often  exceeds  the  resources  of  acquisition 
programs.  Therefore,  program  managers  are  faced  with  the  challenge  of  reducing  a  long  list 
of  potential  alternatives  to  those  that  should  definitely  be  included  in  the  program,  those  that 
should  be  investigated  further  for  potential  inclusion,  and  those  that  should  be  rejected.  The 
integration  of  system  dynamics  and  KVA  provided  a  timely  and  inexpensive  means  of 
evaluating  all  potential  alternatives  and  reducing  the  “long  list”  of  potential  alternatives  to  a 
“short  list”  to  be  pursued  or  investigated  further  based  on  an  objective  and  justifiable 
process.  To  do  this,  first  the  operation  of  the  system  with  each  potential  alternative  is 
simulated  and  the  simulation  model  is  used  to  calculate  the  KVA  productivity  ratios  for  the 
subprocesses  and  system  as  a  whole.  Table  2  provides  an  example  of  a  portion  of  such  an 
analysis  for  the  hypothetical  upgrading  of  the  Predator  UAV  using  the  model  described 
above. 


7  There  are  interdependencies  and  tradeoffs  in  these  alternatives,  such  as  needing  to  increase  the  power  plant 
size  to  maintain  a  given  dash  speed  if  the  fuel  tank  size  is  increased.  These  are  ignored  here  for  simplicity. 
However,  in  application  to  an  actual  program  developers  would  describe  specific  sets  of  features  (e.g.,  possible 
vehicle)  for  analysis. 
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Table  2.  Predator  UAV  Upgrade  Program  KVA  Productivity  Ratios  for  Analysis  of 

Alternatives 


Subprocess  KVA  ratios 

Weapon  System 

Develop 

fire 

mission 

Move 

weapons 

Engage 

targets 

KVA  ratio 

%  Change 
from  Base 

Case 

Predator  Base  Case 

943 

50 

5,094 

705 

0.0% 

Improvement  Alternative 

Increase  fuel  capacity 

100% 

1,886 

50 

5,094 

951 

34.9% 

Increase  fuel  capacity  50% 

1,415 

50 

5,094 

831 

17.9% 

Increase  Power  plant 

100%  for  payload 

849 

50 

7,641 

771 

9.4% 

Increase  Power  plant  50% 
for  payload 

849 

50 

7,641 

771 

9.4% 

Redesign  transmission  for 
100%  faster  dash  speed 

943 

100 

10,188 

741 

5.1% 

Redesign  transmission  for 
50%  faster  dash  speed 

943 

75 

7,641 

727 

3.1% 

Increase  Power  plant 

100%  for  dash  speed 

849 

100 

10,188 

717 

1.7% 

Increase  Power  plant  50% 
for  dash  speed 

849 

75 

7,641 

702 

-0.4% 

Reduce  time  at  base  50% 

943 

52 

5,094 

699 

-0.9% 

Reduce  time  at  base  100% 

943 

51 

5,094 

695 

-1.4% 

The  KVA  productivity  ratios  are  repetitive  for  some  subprocesses  across  alternatives. 
This  is  partially  because  some  alternatives  do  not  change  the  impact  on  some  subprocesses 
and  partially  because  of  the  limited  number  of  system  interactions  incorporated  into  this 
proof  of  concept  model.  However,  the  KVA+SD  modeling  results  are  adequate  to  show  how 
more  accurate  results  might  be  used  in  an  AoA  of  these  potential  capabilities  upgrades. 
Based  on  the  results  above,  a  program  manager  can  assess  the  relative  value-added  of  the 
eleven  alternatives  (including  no-change  as  reflected  by  the  Base  Case)  analyzed. 
Comparison  to  the  base  case  (e.g.,  the  existing  vehicle  in  the  case  of  the  Predator  upgrade) 
provides  an  estimate  of  relative  performance  improvement.  Sorting  the  improvements 
provided  by  potential  alternatives  in  decreasing  order  (Table  2)  lists  alternatives  from  most 
attractive  (Increase  fuel  capacity  100%)  to  least  attractive  (Reduce  time  at  base  50%).  The 
AoA  suggests  that,  if  adequate  resources  are  available,  the  alternative  that  improves  the 
system  the  most  is  to  increase  the  fuel  capacity  100%  because  it  improves  the  development 
of  the  fire  missions.  If  inadequate  resources  are  available  to  implement  this  alternative,  then 


ACQUISITION  RESEARCH  PROGRAM 

GRADUATE  SCHOOL  OF  BUSINESS  &  PUBLIC  POLICY 

NAVAL  POSTGRADUATE  SCHOOL 


733 


the  program  should  attempt  to  increase  the  fuel  capacity  by  50%  for  similar  reasons.  The 
program  manager  can  also  delete  reducing  time  at  the  base  and  a  50%  increase  in  power 
plant  capacity  that  is  used  to  increase  dash  speed  from  consideration  since  they  do  not 
improve  performance.  Certainly,  other  factors  must  be  incorporated  into  a  complete  AoA 
(most  notably  development  costs),  but  the  results  of  the  KVA  analysis  using  the  system 
dynamics  model  provide  valuable  information  for  making  final  recommendations. 

Discussion  and  Conclusions 

Summary 

The  Knowledge  Value  Added  (KVA)  approach  to  including  benefits  in  Analysis  of 
Alternatives  (AoA)  was  integrated  with  a  system  dynamics  model  of  weapon  systems 
operations  to  investigate  the  potential  of  their  integration  to  improve  the  accuracy  of  KVA 
productivity  ratios  and,  thereby,  AoA.  An  integrated  model  was  developed  for  a  generic 
mobile  weapons  system  and  calibrated  to  four  existing  weaponized  Unmanned  Aerial 
Vehicles  (UAV).  Six  basic  subprocesses  of  operations  using  the  weapons  were  included  in 
the  simulation.  KVA  productivity  ratios  for  each  subprocess  for  each  UAV  were  calculated, 
compared,  and  used  to  explain  how  the  simulation  and  KVA  approach  work  together  to 
generate  quantitative  assessments  of  the  relative  value-added  of  each  subprocess  and 
whole  weapon  systems.  A  hypothetical  upgrade  program  to  one  of  the  UAV  was  also 
simulated  to  demonstrate  how  the  integrated  model  can  be  used  to  evaluate  alternative 
upgrades  and  justify  AoA  decisions. 

Evaluation  of  Results 

An  Analysis  of  Alternatives  based  on  an  integrated  system  dynamics/KVA  model 
provides  program  management  teams  with  several  kinds  of  valuable  information. 

Quantified  Measures  of  Improvement  that  include  Benefits:  Measures  of 
subprocesses  and  the  weapon  system  as  a  whole  are  quantified  using  a 
common  set  of  assumptions  and  values  (those  incorporated  into  the 
simulation  model).  Therefore,  differences  in  ratios  and  the  implied  relative 
value  of  different  alternatives  are  due  to  the  differences  in  the  alternatives 
themselves. 

Overall  System  Improvement  Estimates:  The  weapon  (versus  subprocess) 
-productivity  ratios  reflect  changes  in  total  product  operations.  If  adequate 
resources  are  available  to  adopt  at  least  one  alternative,  then  a  list  of 
alternatives  ranked  by  overall  system  improvement  (e.g.,  Table  2)  can  be 
used  to  “triage”  alternatives  into  those  that  should  definitely  be  pursued, 
those  that  require  more  investigation  before  deciding,  and  those  that  should 
be  abandoned.  For  example,  based  on  the  right-hand  column,  Table  2 
suggests  that  increasing  the  fuel  capacity  should  be  pursued  before 
redesigning  the  transmission  and  that  reducing  the  time  at  the  base  should 
not  be  considered  further. 

Guidance  for  Alternative  Selection:  The  analysis  specifically  identifies 
which  alternatives  improve  which  subprocesses  and  the  whole  weapon 
system  and  by  how  much.  For  example,  Table  2  suggests  that  increasing  fuel 
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capacity  increases  the  Fire  mission  development  subprocess  most  and  three 
alternatives  significantly  improve  the  engage  target  subprocess  most. 

•  Justification  of  Analysis  of  Alternatives  Decisions:  When  used  with  the 
simulation  model,  the  KVA  productivity  ratios  can  help  explain  and  justify 
Analysis  of  Alternatives  decisions  by  providing  a  means  of  describing  how 
each  alternative  impacts  operations,  subprocesses,  and  performance.  For 
example,  in  the  UAV  case  above,  increasing  power  plant  size  increases  the 
payload,  which  increases  the  Payload/Lethal  payload  ratio,  which  increases 
the  probability  of  destruction  if  hit  (p(kill)),  which  decreases  the  return  flow 
“Incomplete  kill  rate”  from  the  Battlefield  Assessment  Backlog  to  the  Target 
engagement  backlog  (Figure  3).  This  reduces  the  average  number  of  times 
that  a  target  must  be  engaged  to  be  destroyed,  thereby  improving  the 
productivity  of  the  engage  target  subprocess. 

•  Guidance  for  Further  Investigation:  In  addition  to  suggesting  better  and 
worse  alternatives  to  pursue,  an  integrated  system  dynamics/KVA  model  can 
provide  guidance  for  further  investigation  of  alternatives  by  indicating  which 
subprocesses  each  alternative  improves.  For  example,  Table  2  indicates  that 
the  reason  increasing  fuel  capacity  improves  performance  is  it  improves  the 
Fire  mission  development  subprocess.  The  model  (Figure  3)  indicates  that 
this  occurs  by  increasing  the  vehicle  range,  which  reduces  the  likelihood  of 
losing  a  target  if  it  is  missed  with  ordinance.  Acquisition  program  managers 
can  use  this  information  to  focus  further  investigation  and  development  of  this 
alternative  on  Fire  mission  development  to  assure  that  these  improvements  in 
the  specific  operations  identified  with  the  model  are  realized  during  the 
alternative’s  development. 

It  is  important  to  note  that  neither  a  system  dynamics  model  nor  a  KVA  analysis  of 
this  system  alone  can  reasonably  produce  these  results.  Only  by  integrating  system 
dynamics  and  the  KVA  approach  are  the  benefits  above  available.  Based  on  the  modeling 
and  assessment  above,  we  conclude  that  integrated  system  dynamics/KVA  models  can 
significantly  improve  the  Analysis  of  Alternatives  and,  thereby,  acquisition. 

Implications  for  Practice 

The  current  work  indicates  that  acquisition  can  be  improved  by  using  integrated 
system  dynamics/KVA  models  in  the  Analysis  of  Alternatives.  The  rigorous  development  and 
use  of  integrated  system  dynamics/KVA  models  can  have  important  implications  for 
acquisition  practice,  including: 

The  number  of  alternatives  that  can  be  analyzed  with  KVA  can  be  increased 
due  to  the  relative  ease  of  reflecting  alternatives  in  the  operations  simulation 
model  compared  to  manually  developing  forecasts  for  use  in  KVA  analysis. 
This  increases  the  likelihood  of  identifying  and  selecting  the  optimal 
alternative. 

Justifications  of  AoA  decisions  can  become  stronger  due  to  program 
managers  having  the  ability  to  causally  trace  from  specific  potential  and 
selected  alternatives  through  their  impacts  on  specific  subprocesses  and 
operations  to  performance. 
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•  Justifications  of  AoA  decisions  can  become  more  robust  because  they  can 
reflect  an  analysis  of  a  wider  range  of  alternatives  and  more  alternatives. 

•  Results  of  AoA  can  become  more  consistent  through  the  use  of  a  single, 
integrated  model  of  system  operations  and  KVA  metrics  instead  of  separate 
operations  and  value-added  models. 

•  System  dynamics/KVA  models  may  be  used  to  baseline  product  performance 
during  the  acquisition  process.  Performance  of  the  product  can  be  tracked 
over  time  and  used  to  improve  the  model  and  thereby  performance  forecasts 
and  AoA  later  in  acquisition. 

•  Program  management  will  select  better  alternatives  due  to  the 

implications  for  practice  above.  This  will  generate  more  effective  and 
potentially  cheaper  materiel  solutions. 

In  addition,  improving  AoA  and  acquisition  through  integrated  system  dynamics/KVA 
models  can  improve  CONOPS.  The  Javelin  case  study  described  previously  in  this  report 
provides  a  vivid  example  of  the  ability  of  acquisition  in  general  and  improved  AoA  such  as 
through  integrated  system  dynamics/KVA  modeling  to  impact  tactics  and  strategy.  Upon 
receipt  and  use  of  Javelin,  operators  expressed  surprise  that  its  range  was  twice  that  of  the 
weapon  it  replaced.  That  increased  range  initiated  improvements  to  tactics,  techniques,  and 
procedures  (ttp)  such  as  the  use  of  Javelin  to  detonate  Improvised  Explosive  Devices.  This, 
in  turn,  can  generate  changes  to  strategies.  Accurate  forecasts  of  product  subprocess 
performance  (e.g.,  accuracy  at  longer  range)  can  be  used  to  plan  CONOPS  improvements 
before  product  delivery. 

It  is  important  to  note  that  the  purpose  of  the  simulations  of  operations  developed 
and  illustrated  in  the  current  work  is  to  capture  the  relative  benefits  and  costs  of  different 
materiel  alternatives,  not  to  simulate  the  impacts  of  operations  on  opposing  forces.  The 
usefulness  of  models  can  only  be  judged  in  relation  to  the  specific  purpose  for  which  they 
are  built  (Sterman,  2000).  Therefore,  because  the  purpose  of  integrated  system  dynamics 
and  KVA  models  is  to  improve  AoA,  those  models  should  be  developed,  assessed,  and 
used  separately  from  force-on-force  and  other  simulations  of  operations  developed  for  other 
purposes. 

Future  Work 

The  current  proof-of-concept  work  has  demonstrated  the  potential  of  integrated 
system  dynamics/KVA  models  to  improve  Analysis  of  Alternatives  and  acquisition.  Additional 
research  can  extend  this  work  toward  implementation  and  expanded  application. 
Opportunities  include: 

Modeling  a  specific  acquisition  program  in  support  of  its  Analysis  of 
Alternatives  process  can  develop  and  demonstrate  the  capability  of 
operationalizing  the  approach  tested  here. 

If  important  uncertainties  in  system  operations  are  incorporated  into  the 
system  dynamics  model  it  can  be  used  to  generate  distributions  of  KVA 
productivities.  These  can  be  used  to  estimate  the  volatilities  used  in  real 
options  analysis,  which  has  been  demonstrated  to  be  useful  in  DoD 
acquisition. 
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•  The  application  of  integrated  system  dynamics/KVA  modeling  to  DoD  product 
life  cycle  management  can  be  investigated  by  using  the  model  to  generate 
forecasts  of  performance  and  KVA  ratios  during  acquisition,  comparing  those 
forecasts  with  actual  operations,  and  using  the  results  to  improve  the  model 
fidelity  with  the  system.  The  improved  model  can  then  be  used  to  analyze 
proposed  changes  or  replacement  of  the  system  throughout  its  life  cycle. 

The  Analysis  of  Alternatives  is  a  particularly  challenging  part  of  DoD  acquisition. 
Integrating  system  dynamics  modeling  and  the  Knowledge  Value  Added  approach  has  been 
shown  to  be  capable  of  improving  that  analysis  and,  thereby,  alternative  selection.  Adapting 
this  approach  can  significantly  change  and  improve  DoD  acquisition  practice. 
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Strategy  for  Defense  Acquisition  Research 

The  Software,  Hardware  Asset  Reuse  Enterprise  (SHARE)  repository 


Contract  Management 


Commodity  Sourcing  Strategies 
Contracting  Government  Procurement  Functions 
Contractors  in  21st-century  Combat  Zone 
Joint  Contingency  Contracting 

Model  for  Optimizing  Contingency  Contracting,  Planning  and  Execution 

Navy  Contract  Writing  Guide 

Past  Performance  in  Source  Selection 

Strategic  Contingency  Contracting 

Transforming  DoD  Contract  Closeout 

USAF  Energy  Savings  Performance  Contracts 

USAF  IT  Commodity  Council 

USMC  Contingency  Contracting 


Financial 


Management 


Acquisitions  via  Leasing:  MPS  case 


Budget  Scoring 


Budgeting  for  Capabilities-based  Planning 
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Capital  Budgeting  for  the  DoD 

Energy  Saving  Contracts/DoD  Mobile  Assets 

Financing  DoD  Budget  via  PPPs 

Lessons  from  Private  Sector  Capital  Budgeting  for  DoD  Acquisition  Budgeting 
Reform 

PPPs  and  Government  Financing 
ROI  of  Information  Warfare  Systems 
Special  Termination  Liability  in  MDAPs 
Strategic  Sourcing 

Transaction  Cost  Economics  (TCE)  to  Improve  Cost  Estimates 


Human 


Resources 


Indefinite  Reenlistment 

Individual  Augmentation 

Learning  Management  Systems 

Moral  Conduct  Waivers  and  First-tern  Attrition 

Retention 

The  Navy’s  Selective  Reenlistment  Bonus  (SRB)  Management  System 
Tuition  Assistance 


Logistics  Management 


Analysis  of  LAV  Depot  Maintenance 
Army  LOG  MOD 
ASDS  Product  Support  Analysis 
Cold-chain  Logistics 

Contractors  Supporting  Military  Operations 
Diffusion/Variability  on  Vendor  Performance  Evaluation 
Evolutionary  Acquisition 

Lean  Six  Sigma  to  Reduce  Costs  and  Improve  Readiness 
Naval  Aviation  Maintenance  and  Process  Improvement  (2) 

Optimizing  CIWS  Lifecycle  Support  (LCS) 

Outsourcing  the  Pearl  Harbor  MK-48  Intermediate  Maintenance  Activity 
Pallet  Management  System 
PBL  (4) 


Privatization-NOSL/NAWCI 


RFID  (6) 
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■  Risk  Analysis  for  Performance-based  Logistics 

■  R-TOC  AEGIS  Microwave  Power  Tubes 

■  Sense-and-Respond  Logistics  Network 

■  Strategic  Sourcing 

Program  Management 

■  Building  Collaborative  Capacity 

■  Business  Process  Reengineering  (BPR)  for  LCS  Mission  Module  Acquisition 

■  Collaborative  IT  Tools  Leveraging  Competence 

■  Contractor  vs.  Organic  Support 

■  Knowledge,  Responsibilities  and  Decision  Rights  in  MDAPs 

■  KVA  Applied  to  AEGIS  and  SSDS 

■  Managing  the  Service  Supply  Chain 

■  Measuring  Uncertainty  in  Earned  Value 

■  Organizational  Modeling  and  Simulation 

■  Public-Private  Partnership 

■  Terminating  Your  Own  Program 

■  Utilizing  Collaborative  and  Three-dimensional  Imaging  Technology 

A  complete  listing  and  electronic  copies  of  published  research  are  available  on  our  website: 
www.acquisitionresearch.org 
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System  Dynamics  Modeling  for  Improved 


Knowledge  Value  Assessment: 


A  Proof  of  Concept  Study 


David  Ford  Thomas  Housel  Thomas  Housel  John  Dillard 

Texas  A&M  University  Naval  Postgraduate  School  Naval  Postgraduate  School 
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Introduction  and  Context 

•  “Big  A”  Acquisition  -  initiated  by  JCIDS 

-  Aligned  with  PPBES  -  Top  Down 

-  Needs/Requirements  for  battle  “work” 


CDD 


CPD 


Technology 

Development 


ik 


MSA 


EMD 


me 


ProdS 
Defiloyment 
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JCIDS  Process  and  Acquisition  Decisions 
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Capability-Based  Assessment 

•  CONOPS  are  drivers 

•  DOTMLPF 

•  Often  oriented  upon  costs  (old  “COEA”) 

•  Analysis  of  (Materiel)  Alternatives  =  BCA 


JROC-Approved  Functional  Capability  Boards 


Name 

Sponsor 

Command  &  Control 

USJFCOM 

Battle  space  Awareness 

J- 2 

Logistics 

J-4 

Building  Partnerships 

J-5 

Net-Centric 

J-6 

Force  Application 

J-3 

Force  Support 

J-3 

Protection 

J-8 
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Valuation  of  Systems 


•  Total  Ownership  Costs 

-  Life  Cycle  Cost  Estimate  -  all  colors  of  money 

-  Independent  Cost  Estimates 

•  Key  Performance  Parameters 

-  Given  Ops  Mode  Mission  Summary  Profile 

-  Expected  and  Contingency  scenarios 


•  Simulations  to  address  uncertainty 


-  Monte  Carlo,  Real  options,  Portfolio 
optimization,  etc. 
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Best  “Investment”  Choices 


•  OMB  relies  upon  accuracy  and  completeness 
of  cost  information 


•  credible  cost  estimates  are  vital  for  sound 
management  decision  making  and  for  any  program  or 
capital  project  to  succeed.” 


•  Following  guidelines  in  GAO  Cost  Estimating  and 
Assessment  Guide  will  help  agencies  meet  most  cost 
estimating  requirements.” 


GAO  COST  ESTIMATING 
AND  ASSESSMENT  GUIDE 

IV-i  Practice*  (hi  iMvInvini!  niul  M.mAjUi* 
Capital  Program  C<k(* 
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Intrinsic  Military  Value 


•  Advanced  Anti-armor  Weapon  System  -  Medium: 

-  3  AAWS-M  Technical  Approach  Alternatives 

-  Competitive  Prototypes  in  TD  Phase: 

*  Laser  Beam  Rider 

*  Fiber-Optic  Guided 

*  FUR 


*  Laser  won  the  COEA,  but  EMD  SSEB  chose  the  FLIR 

•  Fire  &  Forget  was  mostly  about  Gunner  Survivability 


Q.:  How  can  Program  Mangers  include  diverse  benefits 
in  Analysis  of  Alternatives? 

A.:  The  Knowledge  Value  Added  (KVA)  Methodology 

•  Measures  values-added  &  costs  of  human  and  IT 
assets  at  sub-process  level  of  operations. 

•  Uses  common  units  of  output  to  aggregate  value- 
added  subprocess  operations  into  numerator  of 
Benefit: Cost  ratio 

•  Uses  common  units  to  aggregate  subprocess 
costs  into  denominator  of  Benefit:Cost  ratio 


KVA  Methodology  Process  Steps 


1.  Identify  core  processes  and  sub-processes. 

2.  Establish  common  units  and  level  of  complexity  to  measure 
learning  time  (knowledge  surrogate). 

3.  Calculate/estimate  learning  time  to  execute  each  sub-process. 
Include  technology  built  into  tools. 

4.  Designate  operation  sampling  time  period  long  enough  to 
capture  representative  sample  of  the  core  processes’  final 
output. 


5.  Describe  benefits  by  multiplying  learning  time  for  each  sub¬ 
process  by  number  of  times  subprocess  executes  during 
sample  period. 

6.  Calculate  cost  to  execute  subprocesses  to  determine  process 
costs. 


7.  Calculate  Return  on  Knowledge  (ROK=  Benefits  or 
Revenues/Cost)  and  ROI  (ROK=  Revenue-Cost/Cost). 
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KVA  Limitations 


•  Estimates  often  from  subject  mater 
experts  -  limits  number  of  alternatives  that 
can  be  considered 


•  Program  managers  often  must  consider 
many  alternatives 

Q.:  How  can  KVA  estimates  of  ROK  be 
improved  to  reflect  more  alternatives  more 
accuracy? 


An  Approach  to  Improving  KVA 
Estimates 


•  A  systems  dynamics  simulation  model  of  the 
subprocesses  that  add  value  and  thereby  generate 
benefits  and  costs 

•  KVA  to  combine  subprocess  benefits  and  costs  into 
common  units  of  measure 

•  Integrate  for  automatic  generation  of  ROK  for  any 
alternative 

•  (Relatively  easily,  quickly,  cheaply)  simulate  multiple 
alternatives  and  ROKs 


•  Analyze  ROK  for  relative  value-added 

•  Use  simulation  model  to  explain  differences  in  ROKs 

The  current  work  sought  to  initially  test  this  approach 
(a  proof-of-concept  study). 
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A  System  Dynamics  Model  of  Mobile 
Weapons  Use:  Moving  Weapons  Sector 


Weapons  leave 

Weapons  at 

base  for  station 

V7 

Base 

A 

Weapons  arrive, 
at  base 


Core  Process 

Mobile  weapons  fleet 
operations 

Moving  weapons  to  &  from 
base  &  station  is  a  value¬ 
adding  subprocess 

Common  units  of  measure 

Benefits:  Time  required  to 
learn  how  to  develop 
technologies  and  skills  to 
move  weapon 

Costs:  Average  time  required 
to  move  weapon 
R5p§i^ 
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A  System  Dynamics  Model  of  Mobile 
Weapons  Use:  Target  Evolution  Sector 
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targets 
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Fire  Support 
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kill 


Confirmed 
target  kills 


Targets 
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Five  more  value-adding  subprocesses 
•  Reverse  flows  describe  four  mission  failure 
modes 


Weapon  capabilities  and  environments  descriptors  include: 

-  p(kill  if  hit)  driven  by  (payload  /  lethal  payload ) 

-  p(hit)  driven  by  (dash  speed  /  target  speed) 

-  p(not  lose  if  miss)  driven  by  (range  /  target  distance  from  base) 
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Example  Results:  Return  on 
Knowledge  Ratios 


Model  was  calibrated  to  four  existing  weaponized  UAV 


Predator 


Simulated  (sub)processes  ,  core  process,  &  KVA  analysis 
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Example 
Model  Use 

•  Hypothetical 
program  to  upgrade 
Predator  UAV  to 
engage  opposing 
UAV  targets 

►  KVA  ROK  that 
reflect  overall 
performance 
improvement 

•  KVA  ROK  that 
reflect  relative 
improvement  in 
individual 
subprocesses 
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Example  Analysis  of  Alternatives: 

Hypothetical  Predator  Upgrade  Program 

•  KVA-based  AoA  indicates  that  increasing  fuel  capacity  100%  Reapei 
improved  overall  performance  most  (34%  weapon  improvement). 

•  If  other  acquisition  constraints  (e.g.  available  funding,  approvals) 
are  met  AoA  recommends  this  alternative. 


Q.:  How  can  the  program  manager  best  explain  and  justify  this 
recommendation? 


-  Increasing  fuel  capacity  by  100%  improves  performance  most  by  improving 
the  “Develop  fire  mission”  subprocess. 


-  From  simulation  model  analysis,  improvement  is  due  to  increasing 
range/target  distance  from  base,  which  reduces  targets  lost  if  missed, 
increasing  opportunities  to  retarget  after  misses. 

•  Analysis  also  identifies  what  subprocesses  to  monitor  during 
development  and  operations  to  assure  and  verify  that  the 
forecasted  improvements  occur. 
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Conclusions 


Analysis  of  Alternatives  based  on  an  integrated  System  X_47B 
Dynamics  /  KVA  model  provides  program  management 
teams  with  several  kinds  of  valuable  information: 

•  Quantified  measures  of  improvement  that  include  benefits 

•  Overall  system  improvement  estimates 

•  Subprocess  improvement  estimates 

•  Guidance  for  alternative  selection  in  Analysis  of  Alternatives 

•  Help  in  justification  of  Analysis  of  Alternatives  decisions 

•  Guidance  for  further  investigation 


Neither  a  System  Dynamics  model  nor  a  KVA  analysis  alone  can 
produce  these  improvements.  Only  by  integrating  System  Dynamics 
and  the  KVA  approach  are  the  improvements  above  available. 


.Acquisition  Research  Program:  Creating  Synergy  for  Informed  Change 


Mcinlrrcy,  <1A 


Impacts  on  Practice 


•  More  alternatives  can  be  analyzed,  therefore  more  robust  AoA  decisions  S^y  Warrior 

•  Stronger  AoA  decision  justifications 

•  More  consistent  AoA  results  -  single,  integrated  model  of  operations  and  KVA. 

•  Better  product  performance  baselines  during  acquisition 

Bottom  Line:  Program  management  will  select  better  alternatives.  This  will 
generate  more  effective  and  potentially  cheaper  materiel  solutions. 


Potential  to  improve  CONOPS:  Javelin  case  study: 

•  Operators  surprised  that  range  was  twice  of  the  weapon  replaced 

•  Increased  range  initiated  improvements  to  tactics,  techniques,  and  procedures  (ttp), 
i.e.  IED 

•  Improved  Javelin  ttp  can  generate  changes  to  strategies 

Accurate  forecasts  of  product  subprocess  performance  (e.g.  accuracy  at  longer 
range)  can  also  be  used  to  plan  CONOPS  improvements  before  product 
delivery. 
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Future  Work 


•  Model  specific  acquisition  program  in  support  of  its  Analysis  of 
Alternatives  process  -  operationalize  the  approach  tested  here. 

•  Incorporate  important  uncertainties  to  generate  distributions  of 
KVA  productivities  and  use  in  real  options  analysis 

•  Improve  product  life  cycle  management 

•Forecast  performance  and  KVA  ratios  during  acquisition 
•  Compare  forecasts  with  actual  operations 
•Use  results  to  improve  the  model  fidelity  with  the  system 
•Use  improved  model  to  analyze  proposed  changes  or 
replacement  of  the  system  throughout  its  life  cycle. 

•  Model  and  analyze  portfolios  of  assets  for  improved  portfolio 
management 
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Questions? 
<  Comments? 
Discussion? 


Related  Research 
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FY07:  Housel,  AEGIS  Platform:  Using  KVA  Analysis,  Risk  Simulation  and 
Strategic  Real  Options  to  Assess  Operational  Effectiveness 
FY08:  Housel  and  Mun,  Potential  Impact  of  Open  Architecture  on  AEGIS  Using 
KVA  and  Real  Options  Analysis 

FY08:  Housel,  Potential  Impact  of  Collaborative  and  Three-dimensional  Imaging 
Technology  on  SHIPMAIN 

FY07:  Dillard  and  Ford,  Modeling  the  Risks  of  Spiral  Development 
FY08:  Dillard  and  Ford,  Modeling  the  Integration  of  Open  Systems  and 
Evolutionary  Acquisition  in  DoD  Programs 
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